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1 INTERACTIVE SCULPTING FOR VOLUMETRIC EXPLORATION 

2 AND FEATURE EXTRACTION 

3 This application is a continuation of PCT/CAOO/00043 filed on January 21 , 2000. 
4 

5 The present invention relates to the field of three-dimensional (3-D) image rendering. More 

6 specifically, the present invention provides a system and method of interactive sculpting for 

7 volumetric exploration and feature extraction. 
8 

9 Background of the Invention 

10 Volume visualization has become increasingly important in clinical practice for the display 

1 1 and analysis of volumetric datasets acquired with imaging methods such as Computed 

f 4 2 Tomography (CT) or Magnetic Resonance Imaging (MRI). Benefits of volume visualization 

fit 3 include the ability to obtain oblique views, the increased understanding of complex geometric 

4 structures; and the abiUty to measure volumes, areas, and distances. Volume visualization 
^;I5 also provides the ability to explore the spatial relationship between an organ and its 

Q16 surrounding structures or tissues. There are two main classes of volume visualization 

- 17 techniques; surface rendering and volume rendering. The present invention relates to volume 

Qt 8 rendering which is a technique that generates a two-dimensional (2-D) projection directly 
from the 3-D volume data without requiring any intermediate geometrical data structure. 

U20 

r: 21 Many different approaches to volume rendering have been proposed over the years. 

22 However, they all suffer to some extent from the problem that volume rendering algorithms 

23 are computationally and memory intensive, although, ever-increasing processor speed, 

24 affordable memory, and clever optimizations have made it possible to perform volume 

25 rendering on low-cost, general purpose personal computers. However, even with interactive 

26 rendering speed, the exploration and visualization of a volumetric dataset is a challenging 

27 task. The increasing amount of data produced by the imaging devices and the occlusion of 

28 the structures of interest by other portions of the subject call for flexible methods to focus the 

29 visualization to a very specific part of the dataset. 
30 

3 1 The classical approach to address these issues has been 3-D anatomical feature extraction by 

32 segmentation. These methods tend to be very time consuming (i.e. slice by slice tracing) or 

33 very anatomy and/or imaging modality specific. A number of researchers have proposed a 
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1 more general semi-automatic segmentation method, however the binary decision process 

2 imposed by the segmentation method often results in loss of information and reduced image 

3 quality. See, for example, Schiemann et al., "Interactive 3D Segmentation", Visualization in 

4 Biomedical Computing 1992, SPIE Vol. 1 808, pp. 376-383, which is incorporated herein by 

5 reference. 
6 

7 An alternative approach is to remove regions with no diagnostic value or regions that obscure 

8 the structure of interest, rather than extracting the structure itself. The classical axis-aligned 

9 cut plane model is often used for this purpose but in many cases, when complicated 

10 morphologies are present, the results are far from optimal. Pflessor et al., "Towards Realistic 

1 1 Visualization for Surgery Rehearsal" First International Conference of Compute Vision, 

12 Virtual Reality and Robotics in Medicine (CVR Med '95), pp. 487-491, April 1995, which is 

1 3 incorporated herein by reference, presents a volume sculpting technique with application to 

14 surgery rehearsal. However, in this technique a segmentation step is required prior to data 

15 manipulation. The technique introduced sculpting tools that perform free-form cutting of 

16 pre-segmented objects. Wang and Kaufman "Volume Sculpting", 1995 Symposium on 

17 Interactive 3D Graphics, pp. 157-156, April 1995, which is incorporated herein by reference, 

1 8 provides a similar technique but with the focus on a modeling tool for the creation of a three- 

19 dimensional object, not on the extraction of specific features from a volume dataset. 
20 

21 It is an object of the present invention to provide a system and method of volume rendering 

22 which overcomes the deficiencies of the prior art. 
23 

24 Summary of the Invention 

25 Accordingly, in a first aspect, the present invention provides a system of extracting a visual 

26 feature from a dataset, comprising a storage means for storing said dataset; retrieval means 

27 for retrieving said dataset from said storage means; display means for displaying an image of 

28 said retrieval dataset; means for defining a block of voxels corresponding to a selected 

29 portion of said displayed dataset, said block containing said visual feature therein; means for 

30 removing from said block voxels not containing said visual feature, to generate a feature 

31 block; and means for generating a mask from said feature block; means for rendering said 

32 dataset using said mask.. 
33 
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1 Interactive sculpting is an advanced technique for volumetric exploration and features 

2 extraction in scalar datasets. Used in conjunction with known fast volume rendering 

3 techniques, the present technique proves to be a powerful and flexible tool especially for a 

4 modality like magnetic resonance imaging (MR), where the underlying physics is not 

5 compatible with the use of opacity transfer functions in volume rendering for tissues 

6 separation. The technique of the present invention is substantially different from classical 

7 segmentation approaches that require a precise definition of the volume of interest. In the 

8 present approach, the sculpted volume does not have to be defined precisely; all the details 

9 are extracted by the volume Tenderer during the proj ection. 
10 

11 In one aspect, the present invention defines two sculpting tools; a jigsaw and an interpolator. 

J.2 

S 3 The jigsaw tool allows the user to "saw-off parts of the volume, i.e., remove regions that are 

% 4 not of interest. The tool is flexible in that it can be applied at any viewing angle and any 

*Ol 5 shape can be drawn which may either include or exclude the region of interest. 

Sl6 

°1 7 The interpolator tool allows the user to roughly outline the region of interest on a few image 

Ol 8 slices and the tool then generates the volume of interest by interpolating between these slices, 

ml 9 The inputs to this tool are inexact regions of interest obtained from manual contouring on 2-D 

~;20 images. 
H21 

22 For both tools, the location and orientation information required by the reconstruction process 

23 can be obtained directly from the display. 
24 

25 In accordance with this invention there is provided a method for extracting a visual feature 

26 from a dataset representing an image of an object, said method comprising the steps of: 

27 displaying said dataset; defining a block of voxels corresponding to a selected portion of said 

28 displayed dataset, said block containing said visual feature therein; removing from said 

29 dataset voxels not contained by said selected portion, to generate a feature block; generating a 

30 mask from said feature block; and rendering said dataset using said mask. 
31 

32 
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1 Brief Description of the Figures 

2 Embodiments of the present invention will now be described, by way of example only, with 

3 reference to the following figures, in which: 
4 

5 Figure 1 is a schematic overview of a system in accordance with the present 

6 invention; 

7 Figure 2 is an illustration of the use of a jigsaw tool in accordance with the present 

8 invention; 

9 Figure 3 is an illustration of the use of an interpolator tool in accordance with the 

1 0 present invention; 

1 1 Figures 4(a) and 4(b) are examples of images generated by the system of the present 

12 invention; 

13 Figures 5(a) and 5(b) are examples of images generated by the system of the present 

14 invention; 

1 5 Figure 6 is a schematic representation of a volume rendering pipeline; 

1 6 Figure 7 is a representation of the pipeline used to create the images of Figure 4(a) 

17 and 4(b); and 

1 8 Figure 8(a) is a flow chart showing the use of jigsaw tool; 

1 9 Figure 8(b) is a flow chart showing the use of the interpolator tool; 

20 Figure 8(c) is a flow chart showing a craniotomy on a MR study according to an 

21 embodiment of the invention; and 

22 Figure 8(d) is a flow chart showing an identification of a pathology in a CTA study. 
23 

24 Detailed Description of a Preferred Embodiment 

25 A system in accordance with the present invention is shown generally at 10 in Figure 1 . 

26 System 10 comprises a processor 12 connected to a user input device 14. User input device 

27 14 may be, for example, a keyboard, a mouse or a touch-screen display. Processor 12 is also 

28 connected to a data storage unit 1 8, and to display means 22. 
29 

30 Data storage unit 18 is used to store pre-acquired 2-D image slices of a subject in digital 

31 form. The stacks of slices are generally referred to as a dataset. The 2-D image slices are 

32 represented by dashed lines in Figure L The 2-D image slices may be acquired using any 
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1 imaging technique, including but not limited to Computed Tomography, Magnetic Resonance 

2 Imaging, and Ultrasound. 
3 

4 Display means 22 displays a rendered volume 26 which is reconstructed from the pre- 

5 acquired 2-D image slices stored in data storage unit 18. The rendered volume displayed in 

6 Figure 1 has a "cut-surface" 28 and a defined 'Volume of interest" (VOI) 30 which will be 

7 explained below. 
8 

9 Prior to describing the method of volumetric exploration and feature extraction of the present 

1 0 invention, the two primary tools of the present invention will be described. 
11 

12 fi) The Jigsaw 

jb 

;gi4 The jigsaw is a versatile cutting tool. The jigsaw can be used in a number of different ways. 

ijl 5 The simplest way is to reduce the effective size of the bounding box by sawing off 

JSjl 6 rectangular regions. In practice, the jigsaw can be used to quickly remove structures that 

Ol7 obscure the anatomy of interest. This method is efficient because careful outlining of 

Ql 8 unwanted structures is not required. It also plays an important role in fast exploration of 

« jjl 9 multiple datasets. 

W20 

u2l When reducing the size of the bounding box is not adequate for removing unwanted anatomy 

22 one could use the jigsaw to generate a cylindrically extruded mask, whose cross section is 

23 defined by an arbitrary 2-D shape. This tool is not restricted to three orthogonal views. In 

24 fact, the direction of extrusion is normal to the plane on which the 2-D shape is defined. In 

25 other words, this tool can be applied from any viewing angle, with 2-D MPR (multi-planar 

26 reformat) or 3-D volume rendering. Furthermore, one could use the jigsaw to either specify a 

27 region to be removed (exclusive) or a region to be visualized (inclusive). This is very useful, 

28 for example, in removing the spine in abdominal studies. 
29 

30 In most cases, the jigsaw is used only to produce a suitable extruded volume but, 

31 occasionally, refinement is necessary. Multiple application of the jigsaw is equivalent to 

32 computing the intersection between two or more extruded masks. Figure 2 illustrates how 

33 applying the jigsaw tool in two different orientations can refine a 3-D shape. The 

5 



1 implementation of the jigsaw is dependant upon the implementation of the Tenderer and the 

2 masking operation. 
3 

4 (ii) The Interpolator 
5 

6 The Interpolator uses a technique known as shape interpolation to generate masks for the 

7 volume rendering operation. Traditionally, shape interpolation has been used to create binary 

8 volume for solid rendering. However, the present approach is to employ this technique for 

9 feature extraction. The inputs to this tool are inexact region(s) of interest obtained from 

10 manual contouring of a few 2-D images (typically, 10 for a dataset with 200 images). The 

1 1 advantage of using this tool is that it is much more time-effective than the painstaking 

12 process of manually outlining the object(s) of interest on every image of the dataset. 
13 

14 The interpolator works by first converting the binary input slices to distance images. In these 

15 images, each pixel value represents the shortest distance between the pixel and the nearest 

16 edge. A positive value indicates that a pixel is inside the region of interest and a negative 

17 value means outside. Cubic spline interpolation is used to compute the missing data in the 

18 gaps, resulting in a series of intermediate slices, which must then undergo a thresholding 

19 operation to produce the final mask. A suitable shape interpolator implementation is 

20 described in S. P. Raya and J. K. Udupa, "Shape-Based Interpolation of Multidimensional 

21 Objects", IEEE Transactions on Medical Imaging, vol. 9, no. 1, pp. 32-42, 1990, incorporated 

22 herein by refernce . 
23 

24 Both the tools discussed above are used to define volumes of interest (VOI). VOI are 

25 irregular volumetric regions that can be used to define the region where the volume rendering 

26 visualization is applied. The VOI does not have to exactly match the structure of interest, but 

27 rather loosely contain it or exclude the obstructing anatomies. The direct volume rendering 

28 process, by using an appropriate opacity transfer function, will allow for the visualization of 

29 all the details contained in the VOI. The main benefit of this type of approach is that the 

30 creation of the VOI is very time-effective, and it can be applied to a variety of different 

3 1 modalities. For example, as shown in Figure 3, to extract the brain in an MR dataset the user 

32 roughly outlines the brain contour on a few slices and then lets the Interpolator tool generate 

33 the VOI. This is particularly significant on MRI datasets because the underlying physics is 
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1 no compatible with the use of opacity transfer functions in volume rendering for tissue 

2 separation. 
3 

4 In most clinical protocols there is the need to extract multiple features, or to separate in 

5 different sections a part of the anatomy. This means that it is preferable to have the ability to 

6 apply multiple VOIs to a single dataset The volume renderer of the present invention solves 

7 this problem in a very general way, by providing the user with the option to define multiple 

8 masks and also with options for assigning a different opacity and color transfer function to 

9 the voxels included in each region. This essentially defines a local classification, where the 

10 color and opacity of a voxel does not only depend on its gray level value, but also on its 

1 1 spatial position. 

ypl3 As discussed above, the brain can be easily extracted from an MR dataset using the 

gl4 Interpolator tool. A second VOI can be created using the jigsaw tool in exclude mode 

fe ffl5 starting from the original dataset. The head and the brain are two distinct VOIs and different 

C316 opacities, color settings, and even rendering modes can be applied to them. In Figure 4 they 

J" 17 are rendered together, allowing one to simulate regular as well as unusual extensive 

J-{18 craniotomy. 

HI 19 

J? 20 In both CT and CTA dataset most of the tissues have distinct Hounsfield values, and so the 

H 21 opacity can be used to selectively visualize a particular tissue type. Nevertheless the 

22 sculpting tools can be used to enhance the visualization, by either applying local 

23 classification to specific anatomical part or to maintain references in the dataset. Figure 5 

24 shows the sculpting techniques applied to both a CT and CTA dataset. In the image shown in 

25 Figure 5(a), the aorta has been identified with the Interpolator tool by drawing very few 

26 contours on the original slices. The rest of the dataset has been partitioned in two regions 

27 using the jigsaw tool. In these three regions different classification has been applied and all 

28 the VOI have been rendered using shaded volume rendering. 
29 

30 In the Figure 5(b) the dataset has been similarly partitioned using the Jigsaw tool. A different 

3 1 classification has been assigned to the two VOI to show soft tissue in one and bone in the 

32 other. Both VOI are rendered in unshaded mode using the original density values. 
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1 A preferred embodiment of the present system has been implement with the Imaging 

2 Applications Platform (LAP) application development toolkit, an object-based software 

3 development package specifically designed for medical imaging applications development. 

4 LAP is designed for visual data processing, and therefore all the important 3-D data structures 

5 are voxel based. The two most important are: binary solid, and gray level slice stack. 

6 Several objects in our visualization pipelines accept both types of data, providing a high level 

7 of flexibility and inter-operability. For example, a binary solid can be directly visualized, or 

8 used as a mask for the volume rendering of a gray level slice stack. Conversely, a gray level 

9 slice stack can be used to texture map the rendering of a solid object or to provide gradient 
10 information for gray level gradient shading. 

11 

J.2 A gray level slice stack (slice stack) is generally composed of a set of parallel cross-sectional 

*03 images (rasters) acquired by a scanner, and they can be arbitrarily spaced and offset relative 

Si 4 to each other. Several pixel types are supported from 8 bit to 16 signed with floating point 

^Jl5 scaling factor. Planar, arbitrary and curved reformats (with user-defined thickness) are 

□16 available for slice stacks. Slice stack can also be interpolated to generate isotropic volumes 

***17 or resampled at a different resolution. They can be volume rendered with a variety of 

Q18 conventional compositing modes such as: Maximum Intensity Projection (MIP) and 

ryl9 unshaded or shaded volume rendering. Multiple stacks can be registered and rendered 

JSpO together, each with a different compositing mode. 
1^21 

22 To generate binary solids, a wide range of segmentation operations are provided: simple 

23 thresholding, geometrical regions of interest (ROIs), seeding, morphological operations, etc. 

24 Several of these operations are available in both 2-D or 3-D. Binary solids can be 

25 reconstructed from slice stack, and shape interpolation can be applied during the 

26 reconstruction phase. Once binary solids have been reconstructed, logic operations (e.g. 

27 intersection) are available together with disarticulation and cleanup functionality. Texture 

28 mapping and arbitrary visibility filters are also available for binary solids. 
29 

30 
31 
32 
33 
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1 The following is a list of relevant objects in this context: 
2 



Object 


Function 


Cproj 


Performs several different types of volume rendering 


Vol 


Compute all the view independent pre-processing 


Ss 


Collects a set of 2-D images (Raster) into a volumetric dataset 


Bv 


Collects and operates on binary volumes (bitvol) 


Recon 


Collects a stack of bitmaps and produces a bitvol 


Siintpm 


Tnternolates resamole and re-mat) a slice-stack 


T im3 


Marmot a set of axis-aliened 3-D dinning narameters 


IjALU V 


Generates extruded or "tubular" bitvol based on one or more irregular regions of interest 


J- VA 


Manage a nixel value mannine 


Prpf 

.TIC! 


A/TanacrpQ nrnpre<5<iive refinement in rendering bv collecting the outnut of multiple 
rendering pathways 


Rf 


Manages the arbitrary reformat of a slice-stack 


Bp 


Performs operations with bitmaps 


Geom2 


Manages 2-D geometry consisting of a list of lines, polygons, arcs and markers 


Tx3 


Manages a 3-D coordinate transformation 


Ras 


Manages the data associated with a 2-D image 



iu 4 A volume-rendering pipeline can be formed by instantiating and connecting few of these 

5 objects. The basic pipeline is shown in Figure 6. 
6 

7 Images that are managed by Ras objects are aggregated to form a stack in the Ss object. 

8 Performing interpolation on the images forms a volume. The Cproj object then renders this 

9 volume, with location and orientation specified by the Tx3 object. Objects that implement 
10 the sculpting tools are normally connected to the Vol object as inputs. 

11 

12 Sculpting with the Interpolator requires that parallel regions of interest (ROI) be specified in 

13 various locations in volume space. To assist the drawing of these ROIs, many parallel cross 

14 sections of the volume have to be exposed to the user. This is achieved by connecting the Ss 

15 object to a Rf object that interpolates the input slice stack onto a cut-surface. The cut- 

16 surfaces can be at any locations and orientations. Moving a cut-surface along its normal axis 

9 
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1 allows the user to seethe various cross sections of the volume. At each cross section, the user 

2 specifies a ROI geometry that will be managed by a Geom2 object. The location and 

3 orientation information required by the reconstruction process can be obtained from the 

4 location and orientation of the viewport. 
5 

6 In the following example, the user draws a number of ROIs that are parallel to the input 

7 rasters. The Geom2 objects send the ROIs in bitmap form to a Recon object, which 

8 constructs a binary volume (bitvol) using the shape interpolation algorithm described above. 

9 When the Vol object receives this bitvol data, it ensures that only parts of the volume lying 
1 0 inside the bitvol are visible. 

11 

12 To construct the bitvol for the jigsaw tool, the Extbv object is used. Similar to a Recon 

!l=l3 object, this object exports a clipping bitvol to a connected Vol object. This object is actually 

04 a meta-object that consists of a combination of Bp and Recon objects. 

3 5 

3 6 The ROI specified by the user is processed by the Extbv object to produce a set of bitmaps 

C|7 (managed by Bp objects) that are used to construct the extruded bitvol. The Extbv object 

CP requires that the orientation and the location of the ROI in volume space be specified. As in 

^19 the Interpolator tool scenario, the information can be obtained by mapping its location and 

y~20 orientation from viewport space to volume space. 
ft 1 

22 The object-based architecture of the Processing Server allows the above sculpting techniques 

23 to be combined to produce a powerful and efficient sculpting tool. This is achieved by 

24 connecting the objects involved in various ways. One way of mixing the sculpting 

25 techniques is to connect both a bitvol exporter (Recon, Extbv) object and a Lim3 object to the 

26 Vol object. In this case only parts of the volume that are inside both the bitvol and the 

27 bounding box are rendered. 
28 

29 A second way to sculpt is to first duplicate a volume, and then applying various combinations 

30 of bitvols, created with the jigsaw and the Interpolator tools, to the two volumes. For 

3 1 example, the image in Figures 4(a) and 4(b) was created by the pipeline shown in Figure 7. 
32 

10 
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1 In this case, the volume is duplicated by connecting the Ss object to two Vol objects. The 

2 two Vol objects are then rendered by the same Cproj object. Each volume is clipped 

3 differently. In the sample image, the head (the lower Vol object) is clipped by a Lim3 object, 

4 exposing the inner volume (the upper Vol object) that is clipped using shape interpolation. 
5 

6 Bitvols can also be created and modified using Bv objects. This object performs various 

7 logical operations such as adding, subtracting and intersecting on input bitvols. The 

8 intersection operation is especially useful for iterative refined clipping regions. 
9 

10 The image in Figure 4 was produced by clipping the outer volume with the complement of 

1 1 the extruded bitvol specified by a ROI. The inner volume was clipped using a mask created 

12 by the interpolator tool. 

li 3 

^14 In the sculpting techniques of the present invention, VOI and classification can be defined 

2l5 interactively in an iterative faction. In this context, the processing speed is an important 

%6 factor, because the user preferably needs to visually assess the result of an operation before 

CI 7 applying the next. Using a progressive refinement approach, the renderer is able to let the 

J^J 8 user interactively analyze a clinical size dataset and, if the results are not satisfactory, the 

2A9 operation can be easily undone and repeated. To allow interactive sculpting, each rendering 

UMO pathway is accompanied by a duplicate that operates on a down-sampled version of the image 
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